Mobile applications with more and more complex graphical user interfaces (GUI) are very popular in the application market today. Test cases which include sequences of user actions and events are usually generated manually to test the GUI's. For example, test scripts are written manually line by line and user actions are recorded case by case. These manual operations are often times tedious and inefficient. Even worse, such manual test generation approaches lack a description of the high-level testing model which explicitly expresses testing purposes during the test generation process. GUI test case generation guided by sketching is a novel approach, in which the testers draw strokes on the screenshots of a mobile application to specify their testing purpose. Then such strokes are identified automatically and constructed into test models, such as Image Based Finite State Machine (IFSM) model. GUI test cases can be generated systematically based on this model. Testers can further improve their test by sketching more strokes on-the-fly if necessary.
As graphical user interfaces have characteristics different from traditional enterprise software, techniques in testing GUI present different challenges and requirements. In the field of mobile application test generation, the state-of-the-art test generation techniques are classified into four categories: random-walk testing, systematic testing, model-based testing and model-learning testing. Random-walk testing, such as Monkey and Dynodroid, generates random sequences of user events to the application under test (AUT). Testing is conducted without any consideration of the actual widgets displayed on the screen, the actual test actions and areas being clicked. Systematic testing, such as Acteve, tests the applications systematically based on symbolic execution technique. It usually collects information from the AUT and combine such information with dynamic symbolic execution technique to generate input events to test mobile applications. Model-based testing, such as MobiGUITAR (a.k.a. AndriodRipper), TEMA, ORBIT and SlumDroid, requires the model of the AUT to generate inputs. The model is usually provided by the tester or obtained by observing and crawling the information of the AUT. Model-learning testing, such as SwiftHand, does not require a complete test model to be provided before the test. Instead, it infers and gains the model on-the-fly while the AUT is under testing. Most existing test generation techniques perform tests either in great reliance on tester's manual efforts, or completely without of tester's control. The manual test, or partial manual test, tends to be expensive in practice. For example, MonkeyRunner requires test scripts to be written by tester manually. Writing scripts manually is a very labor-intensive process, tedious and fallible, and the test cases are often ineffective. TEMA's creation of models also relies on intensive manual work. It requires smaller AUT model components from the tester in order to generate full test model in a parallel composition process. In the automatic test generation process, the entire testing process becomes fully automated without human intervention. Examples of such tools include Monkey, SwiftHand, GUIRipper and A2T 2, etc. Automatic test generation presents new challenges. For example, random-walk testing usually play the entire testing process blindly which results in inefficiencies, systematic testing is often at risk of path explosion, model-based testing and model-learning testing are also affected by state space explosion.
Testing with human intervention will resolve the issues above by combining human tester's powerful intelligence and the machine's enormous computing power. For example, Dynodroid allows interleaving of events from the machine and events from human testers to reach higher source code coverage with fewer test cases. AndroidRipper allows testers to select a sub model manually to avoid state space explosion and generate more accurate test cases in the test generation period. Due to the limitation of the methods themselves, human testers' role is not optimized.
It not only makes tester have a low participation, but also be short of intuition. For example, for the existing model-based test generation techniques, their models often are lack of sufficient expressivity to convey the actual test purpose of tester. Therefore, to tackle these predicaments, we propose a novel sketch-guided test generation technique. A key insight of our approach is to design a practical method to help testers express their test purpose easier and more intuitive. We design an intuitive way to assist testers in expressing their good testing ideas. At the same time, we present an enhanced model called Image-based and Logic-related State Transition Systems (ILSTS) based on state machines, specifically labeled state transition systems (LSTS). By combining human intelligence and ILSTS together, tester can exercise the mobile application more accurate and more targeted. Let's take a look at our test model. In short, in order to show the logical relationship between every different state in one test case, we introduce the first order logic predicates into our test model. For example, the first-order logic predicate ∀, ∃ can be expressed in our model, they can be used as test generation strategy to guide generation work. For our approach, ILSTS brings a more powerful expression ability that can help tester detect more types of bugs. For testers, our approach allows tester generate more accurate test model by sketching. It helps testers (or users) do their testing tasks more visualized and efficient. As mobile devices are extremely convenient in sketching by fingers, testers just need to simply draw a few strokes on the screen to specify their testing purposes. Our framework automatically recognizes these sketches, systematically generates the sequences of events to test AUT based on tester's actual testing purposes. The sketching-based approach extremely alleviate the cost of mobile GUI-based application testing, and generates test cases highly consistent to the intention of users.
FIG. 1A is a screenshot of a graphical user interface on a touch screen, in accordance with some embodiments. The Angry Bird game is used as an example, but the following discussion applies to all other games and applications on a touch screen. In the Angry Bird game, the player shoot birds with different angle and strength in order to destroy pigs and other objects. The screenshot 110 shows an action of shooting by dragging a bird on a slingshot with a specific angle and distance. Different angles and distances result in varying scores in the game. In order to generate high score GUI inputs, the tester needs to specify as many angle and distance patterns as possible and make multiple trials. In traditional test approaches, all of these cases either need to be specified in test scripts manually, or need to be selected by a suitable sub model with significant amounts of labor.
FIG. 1B is an illustration of strokes for testing on the touch screen graphical user interface, in accordance with some embodiments. In comparison to manual testing, under the sketch-guided framework, the tester draws test strokes on the touch screen to perform the test. The screen shot 120 is the GUI of the game in FIG. 1A overlapped with strokes 121, 122 and 123. The strokes are range, logic and action strokes respectively. The details of the range, logic and action strokes will be discussed in details in the paragraphs below. The strokes on the screen provide input to the test system to generate a dragging action starting from the bird on the slingshot and ending at a location in the range specified by the stroke. The stroke 121 in the shape of ∀ is an example of a logic stroke, and the stroke 122 is an example of range stroke. The test system maintains a corresponding universal quantifier constraint as test generation criterion to guide test case generation. The test system automatically produces as many test inputs as possible and the end points of every drag test is in the range defined by the range stroke.